Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update develop-ref after #1916, #1917, and #1919 #1924

Merged
merged 220 commits into from
Sep 17, 2021
Merged

Conversation

JohnHalleyGotway
Copy link
Collaborator

Pull Request Testing

This updates develop-ref to get past changes flagged in NB20210917.

  • Describe testing already performed for these changes:

    The differences are caused by caused by 3 earlier PR's.
  1. PR Feature 1913 MAX_PBL #1916 for Issue Enhance the PBL derivation logic in PB2NC. #1913 changes the number of PBL obs derived:
COMPARING pb2nc/nam.20210311.t00z.prepbufr.tm00.pbl.nc
file1: MET-develop-ref/test_output/pb2nc/nam.20210311.t00z.prepbufr.tm00.pbl.nc
file2: MET-develop/test_output/pb2nc/nam.20210311.t00z.prepbufr.tm00.pbl.nc
ERROR: NetCDF headers differ:
7c7
< 	nobs = 297 ;
---
> 	nobs = 249 ;
  1. PR Feature 1894 pb2nc var all #1917 for Issue Fix PB2NC to better inventory BUFR input data when processing all variables #1894 changes the output for the PB2NC var all unit test:
COMPARING pb2nc/ndas.20120409.t12z.prepbufr.tm00.var_all.nc
file1: MET-develop-ref/test_output/pb2nc/ndas.20120409.t12z.prepbufr.tm00.var_all.nc
file2: MET-develop/test_output/pb2nc/ndas.20120409.t12z.prepbufr.tm00.var_all.nc
ERROR: NetCDF headers differ:
6,8c6,8
< 	nhdr = 14 ;
< 	nobs = 7546 ;
< 	npbhdr = 14 ;
---
> 	nhdr = 16 ;
> 	nobs = 7581 ;
> 	npbhdr = 16 ;
10c10
< 	nhdr_sid = 5 ;
---
> 	nhdr_sid = 6 ;
13c13
< 	obs_var_num = 122 ;
---
> 	obs_var_num = 163 ;
  1. PR Feature 1858 obs quality #1919 for Issue Add Point-Stat and Ensemble-Stat obs_quality_exc configuration option to specify which quality flags should be excluded #1858 renames and adds output related to the obs quality control flags:
ERROR: folder MET-develop-ref/test_output missing 9 files
    ensemble_stat/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_ecnt.txt 
    ensemble_stat/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_ens.nc 
    ensemble_stat/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_orank.txt 
    ensemble_stat/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V_rhist.txt 
    ensemble_stat/ensemble_stat_QTY_INC_EXC_MADIS_VGS_20120409_120000V.stat 
    point_stat/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V_ctc.txt 
    point_stat/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V_mpr.txt 
    point_stat/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V_sl1l2.txt 
    point_stat/point_stat_QTY_INC_EXC_PB_120000L_20120409_120000V.stat 
ERROR: folder MET-develop/test_output missing 16 files
    ensemble_stat/ensemble_stat_QTY_MADIS_VGS_20120409_120000V_ecnt.txt 
    ensemble_stat/ensemble_stat_QTY_MADIS_VGS_20120409_120000V_ens.nc 
    ensemble_stat/ensemble_stat_QTY_MADIS_VGS_20120409_120000V_orank.txt 
    ensemble_stat/ensemble_stat_QTY_MADIS_VGS_20120409_120000V_phist.txt 
    ensemble_stat/ensemble_stat_QTY_MADIS_VGS_20120409_120000V_relp.txt 
    ensemble_stat/ensemble_stat_QTY_MADIS_VGS_20120409_120000V_rhist.txt 
    ensemble_stat/ensemble_stat_QTY_MADIS_VGS_20120409_120000V_ssvar.txt 
    ensemble_stat/ensemble_stat_QTY_MADIS_VGS_20120409_120000V.stat 
    point_stat/point_stat_QTY_PB_120000L_20120409_120000V_cnt.txt 
    point_stat/point_stat_QTY_PB_120000L_20120409_120000V_ctc.txt 
    point_stat/point_stat_QTY_PB_120000L_20120409_120000V_cts.txt 
    point_stat/point_stat_QTY_PB_120000L_20120409_120000V_eclv.txt 
    point_stat/point_stat_QTY_PB_120000L_20120409_120000V_fho.txt 
    point_stat/point_stat_QTY_PB_120000L_20120409_120000V_mpr.txt 
    point_stat/point_stat_QTY_PB_120000L_20120409_120000V_sl1l2.txt 
    point_stat/point_stat_QTY_PB_120000L_20120409_120000V.stat 
  • Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

    None

  • Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]

  • Do these changes include sufficient testing updates? [Yes]

  • Will this PR result in changes to the test suite? [Yes]

    If yes, describe the new output and/or changes to the existing output:

    Update reference version, as expected.

  • Please complete this pull request review by [Fill in date].

Pull Request Checklist

See the METplus Workflow for details.

  • Complete the PR definition above.
  • Ensure the PR title matches the feature or bugfix branch name.
  • Define the PR metadata, as permissions allow.
    Select: Reviewer(s)
    Select: Organization level software support Project or Repository level development cycle Project
    Select: Milestone as the version that will include these changes
  • After submitting the PR, select Linked issues with the original issue number.
  • After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
  • Close the linked issue and delete your feature or bugfix branch from GitHub.

jprestop and others added 30 commits June 14, 2021 12:24
Commented out items and added text to description
…to_laton

Bugfix 1817 point2grid latlon to laton
…arsing

Bugfix 1508 tc_gen file list parsing
…_double_type_lat_lon

Bugfix 1838 point2grid supports the double type lat/lon
* Per #1833, changed references to met_help to Discussions.

* Per #1833, fixed typo and removed an unnecessary word
Howard Soh and others added 23 commits September 9, 2021 17:03
* Per #1912, update logic of Met2dDataFile::process_data_plane(). Rather than returning bad status for a field of all bad data, print a warning message and continue processing.

* Per #1912, fixing typo in error message that I ran into. Doesn't actually apply to this issue.

* Per #1912, making an unrelated change here to fix up the MODE-Analysis log output. It was reporting (nul) for the number of lines being read/kept. Simplify the log messages to print the correct numbers.
* Per issue #1858: changed parse_conf_obs_qty to parse_conf_obs_qty_inc. Added new function parse_conf_obs_qty_exc. SL

* Per issue #1858: changed conf_key_obs_qty to conf_key_obs_qty_inc. Added conf_key_obs_qty_exc. SL

* Per issue #1858: modified vx_pd.set_obs_qty_filt() to use parse_conf_obs_qty_inc() instead of parse_conf_obs_qty(). SL

* Per issue #1858: added back the original parse_conf_obs_qty() function for backward compatibility. SL

* Per issue #1858: added back config_constants.h for backwards compatibility. SL

* Per issue #1858: added code to read in obs_qty_inc instead of obs_qty. Also added in code for backwards compatibility to check for obs_qty (obs_quality) and use that for obs_qty_inc. SL

* Changed obs_qty_filt to obs_qty_inc_filt. Added obs_qty_exc_filt, including relevant function. SL

* Per issue #1858 added vx_pd.set_obs_qty_exc_filt. SL

* In add_point_obs() added if block to apply obs_qty_exc_filt. SL

* Per issue #1858, in add_point_obs(), cleaned up sections that apply obs_qty_inc and obs_qty_exc filters. Fixed spacing issue. SL

* Per issue #1858 in process_config(), cleaned up sections that set obs_qty_inc_filt and obs_qty_exc_filt. Fixed spacing issue. SL

* Per issue #1858 in process_obs_file() cleaned up code. SL

* Per issue #1858: changed obs_qty_filt to obs_qty_inc_filt. Added obs_qty_exc_filt. Modified add_point_obs() to use obs_qty_exc_filt. SL

* Per issue #1858: in process_config(): added code to set_obs_qty_inc_filt and set_obs_qty_exc_filt. Also added code to check if old obs_quality is being used, if so, set it to obs_qty_inc_filt. SL

* Per issue #1858: in add_point_obs() modified section that applies the obs_qty_inc and obs_qty_exc filters. Combined the if-blocks into one concise if-block that applies both filters. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_inc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_exc. SL

* Per issue #1858: For both the point_stat and ensemble_stat tests: modified tests that use OBS_QTY to use OBS_QTY_INC. Added unit test for OBS_QTY_EXC (obs_quality_exclude). SL

* Per issue #1858: just added comment at top of file indicating what changed. SL

* Per issue #1858: just added comment at top of file indicating what changed. SL

* Per issue #1858: removed parse_conf_obs_qty. Modified parse_conf_obs_qty_inc to check for old 'obs_quality' entry. SL

* Per issue #1858: in process_config(), modified section that calls set_obs_qty_inc_filt...this new function can now handle old 'obs_quality' entry as well as the new 'obs_quality_inc'. SL

* Modified parse_conf_obs_qty_inc and parse_conf_obs_qty_exc. SL

* Per issue #1858: re-updated parse_conf_obs_qty_exc and reverted back to the simplier version of this function. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_exc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_exc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_exc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_exc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc and added obs_quality_inc. SL

* Per issue #1858: replaced obs_quality with obs_quality_inc. Added documentation for obs_quality_exc. SL

* Per issue #1858: checking in latest version. Modified section that calls set_obs_qty_inc_filt...this new function can now handle old 'obs_quality' entry as well as the new 'obs_quality_inc'. SL

* Per issue #1858: cleaned up some extra line spaces. SL

* Per #1858, removed the deprecated obs_quality configuration entry from 3 EnsembleStat and 5 PointStat configuration files.

* Per issue #1858: duplicated fcst/obs fields to run/test with both obs_quality_inc and obs_quality_exc. SL

* Per issue #1858: duplicated fcst/obs fields to run/test with both obs_quality_inc and obs_quality_exc. SL

* Per issue #1858. Deleted these files git repository. These were replaced with: PointStatConfig_qty_inc_exc. SL

* Per issue #1858. Deleted these files from the git repository. These were replaced with: EnsembleStatConfig_qty_inc_exc. SL

* Per issue #1858. Deleted these files from the git repository. These were replaced with: PointStatConfig_qty_inc_exc. SL

* Per issue #1858: Modified: Don't need to run separate tests for obs_quality_inc and obs_quality_exc, these are now combined into one test via the updated config files: PointStatConfig_qty_inc_exc and EnsembleStatConfig_qty_inc_exc. SL

* Per #1858, update the logic in parse_conf_obs_qty_inc() to use last_lookup_status(). See forthcoming GitHub issue comment about this.

* Per #1858, tweak the warning message.

* Per #1858, made the existing unit_quality_filter.xml tests a bit more concise. Call Point-Stat and Ensemble-Stat only once each time. But for each, test 3 things: all qty, obs_quality_inc, and obs_quality_exc.

* Per #1858, remove the _ON from the job name since the _OFF has been included in the same configuration.

Co-authored-by: Seth Linden <linden@kiowa.rap.ucar.edu>
Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu>
* #1894 Added station ID 14008 to pb2nc_NDAS_var_all unit test

* #1894 Check the existance of the valid data up to 16 levels

Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu>
* #1913 Changed MAX_PBL to 10000. Filter out if the PBL input data is invalid

* #1912 Replaced is_eq to is_bad_data

* #1913 Changed is_eq to is_bad_data

Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu>
…phasis prior to the METplus Governance meeting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment